Evaluation of a Continuing Professional Development program for first year student pharmacists undergoing an Introductory Pharmacy Practice Experience

Objectives: The purpose of the study was to evaluate a live and online training program for first year pharmacy students in implementing Continuing Professional Development (CPD) principles (Reflect, Plan, Act, and Evaluate), writing SMART learning objectives, and documenting learning activities prior to and during a hospital introductory professional practice experience. Design: Cohort Study. Setting: Introductory professional practice experience. Participants: First year (PY1) students at the University of North Carolina Eshelman School of Pharmacy. Intervention: Live training or online training to introduce the concept of Continuing Professional Development in practice. Main Outcomes: Implementation of CPD principles through 1) completed pre-rotation education action plans with specific, measurable, achievable, relevant and time-bound (SMART) learning objectives; and 2) completed learning activity worksheets post-rotation indicating stimuli for learning, resources used and accomplished learning. objectives; and 3) documented suggestions and content feedback for future lectures and pharmaceutical care lab experiences. Results: Out of the whole cohort (N=154), 14 (87.5%) live (in person) trainees and 122 (88%) online trainees submitted an education action plan. Objectives were scored using a rubric on a scale of 1-5. A rating of 5 means “satisfactory”, 3 means “work in progress” and 1 means “unacceptable”. There were significant differences between the mean live trainee scores and the mean online trainee scores for the following respective section comparisons: Specific 4.7 versus 3.29 (p<0.001); Measurable 3.9 versus 2.05 (p<0.001); number of objectives 3.6 versus 4.6 (p<0.001); and average grade 92.9 versus 77.7 (p<0.001). Of the 396 learning activity worksheets reviewed, 75% selected discussion with peers and/or health providers as a stimulus for learning. Students reported spending an average of 50.2 hours completing the learning objectives. All of the pre-stated objectives were fulfilled completely or partially. Conclusion: Live trainees performed significantly better than online trainees in writing SMART learning objectives. With focused training, students are more capable of implementing principles of CPD. Introduction Maintenance of knowledge, skills and abilities after graduation is vital. In pharmacy, this has traditionally been achieved by way of continuing education. Continuing professional development (CPD) is an alternative approach and attempts to make this career path a lifelong learning process. The idea of CPD is being discussed as a model for pharmacists in the United States. 1 The Accreditation Council for Pharmacy Education (ACPE) defines CPD as “the lifelong process of active participation in learning activities that assists individuals in developing and maintaining continuing competence, enhancing their professional practice, and supporting achievement of their career goals”. 2 Others have defined CPD as an ongoing, self-directed, structured, outcomes-focused cycle of learning and personal improvement. 1 Original Research EDUCATION http://z.umn.edu/INNOVATIONS 2011, Vol. 2, No. 2, Article 40 INNOVATIONS in pharmacy 2 The CPD cycle in this study uses a five-stage cycle adapted with permission from the Ontario College of Pharmacists. 1, 3, 4 The four main principles of this model include: Reflect, Plan, Act, and Evaluate. A fifth aspect, Record and Review, is an integral aspect of the other four parts. CPD includes both structured learning, relating to typical continuing education courses, and unstructured learning, which can come from various sources such as drug information questions posed by patients or healthcare providers. This allows specific professional development tailored towards the pharmacist’s own desired learning objectives. Although relatively new to the profession of pharmacy in the United States, CPD is currently being used in Canada, the United Kingdom, New Zealand and Australia as a widely accepted means of ongoing learning by pharmacists. 1,4,5-9 Data reflecting the US experiences are also available and suggest benefit to practicing pharmacists. 10,11,12 While CPD has been studied in the practicing pharmacist population, it has rarely been examined among pharmacy students. The Accreditation Council for Pharmacy Education (ACPE) has endorsed self-directed learning for students and supports the maintenance of a performance portfolio in their guidelines and standards. 13 Introducing CPD to students early on in their curriculum would adhere to that standard. It has been suggested that to increase accuracy of students’ selfassessment skills, frequent feedback must be given along with verification of specific self-assessment surveys. 14 Therefore, careful planning and training must take place to increase the success of CPD for students. The University of Manchester School of Pharmacy, United Kingdom, has introduced final-year students to reflective CPD practice for two years. 15 Their results suggest more emphasis needs to be given to the value of CPD early in pharmacy education curricula. Similarly, Wallman and colleagues (2008) set out to assess pharmacy student’s levels of reflection during internships. They point out that reflective skills should be introduced to students early in their pharmacy education since the goal of pharmacy education is to give students a good knowledge base as well as motivation and ability for continuous professional development. 16 In 2010, Janke acknowledged that CPD studies have been focused on practicing pharmacists thus far. 17 She suggests that introducing CPD into academics will help new practitioners transition to the CPD model for lifelong learning. She states “Academics must align interests in self-directed learning and lifelong learning with the profession’s current discussions on CPD. Then, we must support students in developing CPD skills while in school.” She also recognizes experiential learning as the ideal place to practice and reinforce CPD. 17 In the delivery of key concepts to a large audience, online training is very desirable by many because of its flexibility, ease of access, and low cost. Several studies examining the effectiveness of online continuing medical education (CME) have yielded positive results. A study compared an internetbased CME that could be completed over two weeks to a single, live, small-group interactive workshop. 18 Its results revealed that both online and live groups showed similar short and long-term knowledge gains in the subject matter. 18 Sly JL and colleagues had similar results that electronic CPD is a suitable delivery method when they piloted a web-based education program to primary care physicians in Australia and Italy. 19 Online training requires independent learning that may not suit everyone. Live training programs can offer benefits over online training by providing face-to-face, guided instruction and ability to answer questions as they arise. This was demonstrated in studies evaluating students in online versus live learning programs. In a study of pre-clinical dental students, oral lectures got better educational and enjoyment survey values than a computer assisted learning group. 20 An assessment of pharmacy students in Canada revealed face-toface interactions with peers and instructors were rated higher than online interactions in a pharmacokinetics course using blended learning methods. 21 To date, there are no US studies that have evaluated the role of a training program in helping pharmacy students write learning objectives and implement a CPD approach early on in the pharmacy curriculum. This study fills this gap in research by examining the role of a CPD training program (online and live) in helping first year pharmacy students write SMART (Specific, Measurable, Achievable, Relevant and Timed) learning objectives and implement a CPD process as they progress through experiential training. Objectives The purpose of the study was to evaluate a live and online training program for first year pharmacy students in implementing CPD principles (Reflect, Plan, Act, and Evaluate), writing SMART learning objectives, and documenting learning activities prior to and during a hospital introductory professional practice experience. Methods This project was approved under exempt status and a waiver for Informed Consent was granted from the UNC IRB as the data under study was already a component of students’ curricular requirements at the UNC Eshelman School of Pharmacy. The study cohort of 154 students included all UNC Eshelman School of Pharmacy students enrolled in the first didactic Original Research EDUCATION http://z.umn.edu/INNOVATIONS 2011, Vol. 2, No. 2, Article 40 INNOVATIONS in pharmacy 3 professional year. They attended a one-hour live classroom CPD lecture session. The following content areas were discussed: definition of CPD, CPD process, the North Carolina CPD story, lessons learned so far, strategy for spread and advantages of using CPD to maintain competence in pharmacy school. The proposed project was introduced after the lecture and students were asked if they would volunteer for the live CPD group. All 154 students were instructed to complete a CPD 101 webcast. 2 The webcast was developed by ACPE and made available on the ACPE website. CPD 101 describes the concepts and components of CPD, the need to improve how pharmacists approach their continuing education and lifelong learning, and compares and contrasts the traditional CE model with a CPD model. A group of 16 students volunteered to attend a live condensed CPD training. 3,12 The live condensed training consisted of a 3-hour lecture and discussion with active learning principles. At the end of the program the participants were expected to be able to: 1) Review the CPD process and learning plan; 2) Refine their learning objectives, learning plan and overall CPD process; 3) Discuss an effective documentation plan for learning activities; and 4) List tips for successfully implementing CPD in their practice site. The other 138 students were directed to listen to three additional prerecorded CPD webcasts online at the ACPE website, 2 which took approximately three hours to complete. The three additional webcasts included 1) Inventory of learning styles (describes the role of self-reflection in CPD and how to apply learning styles theory to facilitate selfreflection), 2) Using reflection to create a learning plan (describes the importance and application of reflection in personal and professional development and how to design learning objectives that address the personal and professional goals identified through reflection), and 3) Act, Evaluate and Record your CPD (describes how to implement a personal learning plan to accomplish identified learning objectives, evaluate your learning and overall CPD process and to develop and maintain a CPD portfolio). The live training focused on all three with less detail on the learning styles. The main difference between the two training modalities was: 1) the interactive nature of the live session, and 2) the presence of two faculty mentors to help provide feedback on initial SMART learning objectives provided by the students. Since we had successfully used the condensed CPD training model to train pharmacists, the investigators were curious whether the same successes would translate into an all student cohort. 3,12 At this time we were also faced with the potential challenge of training 154 students in small sessions over a period of time before their summer rotations. This was our rationale to utilize the available webcasts on the ACPE website and evaluate whether the outcomes would be different for the live and online trainees. The investigators believed that the hospital introductory pharmacy practice experience (H-IPPE) would be enhanced if students used the CPD process to prepare their own learning goals and objectives and to better prepare them for their first early practice experience. The school’s H-IPPE is a monthlong experience that takes place in the summer following the first professional year. The primary intent of the H-IPPE is to facilitate students’ continuing professional development in the context of the hospital pharmacy practice setting. Through structured activities and assignments, students build upon knowledge and skills developed in the first year of the didactic curriculum. Students continue to explore the concepts of professionalism and shared accountabilities for health care outcomes; formulate a personal philosophy of and approach to professional practice; expand drug and disease knowledge; and develop practical, critical thinking and life-long learning skills. The H-IPPE was also considered a good introduction point for CPD into the pharmacy curriculum. Students submitted their CPD portfolios and SMART objectives to Blackboard®, the school’s course management software, before their H-IPPE. The portfolio included a portion for reflection on past lectures and practice, an education action plan (Appendix A) complete with SMART learning objectives, their learning activity worksheets (Appendix B) and a section for recommendations for future lectures and pharmaceutical care lab activities. Electronic post-training feedback surveys were deployed to the study cohort after completing their H-IPPE. Education action plans, showing their SMART learning objectives for the rotation, and the learning activity worksheets (LAWs) were downloaded from Blackboard® by the primary investigator for evaluation and summary. Summary data did not contain any student identifiers. Study methods included the evaluation of elements from the Education Action Plans (Appendix A) and the Learning Activity Worksheets (LAWs) (Appendix B). The students used the education action plan to summarize their learning objectives prior to the H-IPPE rotation. They were instructed to submit this plan prior to the start of their H-IPPE rotation. A scoring/grading rubric (Table 1) was used to assess students’ education action plans and was modified from the published rubric at the Round World Media website. 22 The rubric employs performance ratings of satisfactory, work in progress and unacceptable. Objectives were considered “SMART” if they were Specific, Measurable, Achievable, Relevant and Original Research EDUCATION http://z.umn.edu/INNOVATIONS 2011, Vol. 2, No. 2, Article 40 INNOVATIONS in pharmacy 4 Timed. 17 An example of a “SMART learning objective is “By June 2011, Compare and contrast antihypertensive drugs used to treat hypertensive emergencies in the pregnant patient.” Points for each of the sections (Specific, Measurable, Achievable, Relevant and Timed) were added up to a total score of 25 and then converted to a grade of up to 100 possible points. These grades were then tabulated and analyzed using descriptive statistics (Mean, standard deviation and variances). Means were calculated using Microsoft Excel data management software. Comparison between the live group and the online group means were done using the students’ two tailed T-Test with Microsoft Excel data management software. Statistical assumptions for using the t-test were met prior to using this approach. A preliminary test for the equality of variances indicated that the variances of the live group and the online group was significantly different for the rubric scores under the column marked “Specific” (p=0.001); therefore, a two-sample t-test was performed for this comparison instead. A Wilcoxon Rank Sum test was also performed for this comparison with the null hypothesis indicating that the two samples came from the same populations (In other words, there is no difference between the 2 groups). All the other scores had an equal variance using the preliminary test therefore we proceeded with the t-test that assumes equal variances. Learning activity worksheets (Appendix B) were submitted electronically and anonymously upon rotation completion. The worksheets that we used for the study were adapted for student purposes from the NC CPD portfolio which were adapted and revised with permission from the Ontario College of Pharmacy CPD portfolio. 3 The following were collated and summarized: 1) stimulus of their learning objectives; 2) resources used to complete objectives; 3) time spent on learning objectives; and 4) suggestions for upcoming lectures and lab classes. Students could select more than one option in the first two categories. The number of times a student used a stimulus or resource was noted and the number of total stimuli and resources that each student used was also noted. Time spent on learning objectives documented on the LAWs, were counted and averaged as well. The students were asked to respond to the following statements in the portfolio after completing the HIPPE rotation: 1) Based on my experience so far, the following activities in the upcoming PCL will enhance my knowledge and abilities; 2) Based on my experience so far, emphasis on the following topic areas in future lectures will enhance my knowledge and abilities. A list of suggestions for upcoming PCLs and lectures were then collated and summarized. Results Learning Objectives A sample of the students’ learning objectives is listed in the Education Action Plan in Appendix A. Of the study cohort (154), 136 (88.3%) submitted an Education Action Plan by the deadline; 14 (87.5%) live trainees and 122 (88%) of the online trainees. The live training resulted in 72% of students with a grade of > 90, and 14% each with grades between 80-89 and 70-79 (Figure 1). There were no students with grades <70 among the live trainees. Online training brought about 29% of students with a grade of > 90, 18% with grades between 8089, and 17% with grades between 70-79, and 36% students with grades <70. When both groups were combined, 34% of students had a grade of > 90, 17% each had grades between 80-89 and 70-79, and 32% students had grades <70. The average (mean) scores for each of the sections (Specific, Measurable, Achievable, Relevant and Timed) are shown in Table 2. The students in the live group were different from those in the online group in that they volunteered to participate in the live training whereas the online group had to participate in order to meet H-IPPE course requirements. Demographics were not collected to preserve anonymity. There were significant differences between the mean live trainee scores and the mean online trainee scores for these aspects of the SMART learning objectives: Specific; 4.7 vs. 3.29 (p<0.001), Measurable: 3.9 vs. 2.05 (p<0.001). Using the Wilcoxon Rank Sum test, the absolute Z score (50.74) for the “Specific” data was greater than the critical Z score (1.96) for a two-tailed test with a level of significance of 0.05. We therefore rejected the null hypothesis and are 95% confident that the 2 groups are different. There was no significant difference between the two groups when comparing the Achievable, Relevant, and Timed elements of SMART objectives. There were significant differences between the mean live trainee scores and the mean online trainee scores in these two areas: number of objectives [3.6 vs. 4.6 (p<0.001)] and average grade [92.9 vs. 77.7 (p<0.001)]. All trainees created between one to six objectives with most students stating five objectives. As expected, most of the objectives were related to the upcoming rotation. Some were associated with more long term goals such as giving presentations and communication with patients and professionals (Appendix A). Learning Activity Worksheets Data from a total of 396 worksheets submitted by 120 students were reviewed. Each student submitted one to six worksheets each. The average number of worksheets submitted by each student was 3.3. One stimulus for learning was identified on 35% of the learning activity worksheets; 65% identified at least two stimuli for learning. As shown in Original Research EDUCATION http://z.umn.edu/INNOVATIONS 2011, Vol. 2, No. 2, Article 40 INNOVATIONS in pharmacy 5 Table 3, discussion with peers and/or health providers was selected as a stimulus for learning on 75% of the worksheets, while 35% chose managing a patient or practice problem. An example of a patient problem was considered to be a patient case that dealt with treating specific disease states whereas a practice problem example was a student working on an IV preparation exercise. Up to two resources were used on 67% of worksheets. As shown in Table 4, common learning resources used were rotation activities (82%) and literature evaluation (35%). The mean amount of time spent on completing the stated learning objectives was 50.2 hours. 53% of the pre-stated objectives were entirely fulfilled during the rotation and 47% were partially fulfilled. The fulfillment of objectives at this rotation was not compared to fulfillment of the objectives at other sites or in the classroom. There was no definition given for entirely or partially fulfilled. We relied on the student’s perception of how much they learned. The most common suggestions for upcoming lectures were: 1) pharmacotherapy modules, 2) infectious diseases 3) pharmacokinetic topics, 4) hematology/oncology, 5) healthcare communication skills, 6) interpretation of lab values, 7) pharmacy laws and procedures, 8) discussion about careers in pharmacy, 9) cause of drug interactions/pathways for interactions, and 10) how to properly make IVs. Discussion SMART learning objectives For the whole class, the section on the SMART objectives with the least average score was “Measurable” followed by “Specific” (Table 2). This finding mirrors results of a study of the CPD process with practicing pharmacists also completed in North Carolina. 3,12 While training pharmacists, more difficulty was observed in choosing appropriate measurable verbs for the objectives than the other sections in SMART. The better performance in the live trainee group may have been due to extra attention to coach them around writing appropriate learning objectives. We were also unable to ascertain how many of the online trainees actually completed the pre-specified modules from the ACPE website (ACPE was still working on tracking capabilities at the time of the study). We considered scheduling live training sessions and workshops for incoming first year students (N=154) given the results of this study. The challenges it presented for multiple small group sessions, resulted in the use of online CPD training modules for current student cohorts. To improve learning outcomes and minimize potential gaps with the online training modules, focused live training with particular emphasis on how to write SMART learning objectives targeting the specific and measurable components will be conducted for the students who completed the online sessions. Students enrolled in CPD in the future will also be coached specifically on writing SMART objectives. Though there are data investigating CPD in students in other countries, we did not find data specific to how well first year students write SMART learning objectives and so we were unable to compare our results with other data. We plan to evaluate the cohort of students’ scores after the follow up training on writing SMART learning objectives. In developing programs for students and pharmacists on CPD, it is essential to spend sufficient time on the aspect of developing SMART learning objectives. We also noticed that the main motivation for students is a grade and as such presents challenges in having them return their portfolios in an honors system. The use of electronic portfolios helps in this regard and was also a recommendation in a CPD program to pharmacists. 3,12 In the CPD study among fourth year UK pharmacy students, half of the students were asked for their opinions of the process halfway through the 12-week program. Students who completed the final evaluation survey “thought reflective practice to be a good idea in theory (66%), but not in practice (79%)”. The workbook given to the students was considered to be tedious (99%) and time consuming (88%). 15 As we found in our population, this further reinforces the importance of using a process that is easy to follow and apply, for the students to ensure desirable outcomes and program success. Feedback from the students in our study was mostly positive in that regard. Learning activity worksheets We noted that peer discussion played a stronger role for selecting learning activities than either patient care or practice problems. This may be suggestive of the novice learner who completes assignments according to instructor requirements and feedback from peers. Further research includes finding out the advanced learner’s approach, and main stimulus to learning. It was also not surprising that the two main stimuli for learning were the rotation activities and literature evaluation. As the learner advances, we expect that a more self-directed approach will emerge which will hopefully reflect in their learning objectives as they progress through pharmacy school. CPD-related student growth was show in a study on pharmacy interns at Uppsala University, Sweden. Students wrote reflective essays about patient counseling at the start and the end of their internships. Based on the essay scores, there was a significant increase in the number categorized as reflective when compared to baseline (25% vs. 60% p<0.001). 16 In order to keep the learning and CPD cycle ongoing between the rotations and the didactic portions of the curriculum, we solicited feedback from the students on future learning Original Research EDUCATION http://z.umn.edu/INNOVATIONS 2011, Vol. 2, No. 2, Article 40 INNOVATIONS in pharmacy 6 experiences. This was to help begin the reflective process during the didactic and lab components of the pharmacy curriculum and foster CPD during those segments of learning. The students’ list of potential future pharmaceutical care lab and lecture topics were submitted to the Director of Pharmaceutical Care Labs and Director of Experiential Education to inform future curricula planning and maximize flow between the didactic and experiential components of the curriculum. Several studies have suggested the incorporation of CPD early in the curriculum to make it easier for practicing pharmacists to adopt lifelong learning using CPD principles. 14-17 This study is a step in that direction for the pharmacy profession. Faculty and preceptors can reinforce CPD principles among the students as they progress through the curriculum. Our hope is that as they become practicing pharmacists, the process will be more automatic as they identify practice gaps and take the initiative to design education action plans to grow accordingly. Limitations The main limitation of this study is that we had no way of determining how many of the students actually completed the online modules on the ACPE website since electronic tracking was not available at the time. The variances observed in each group were likely unequal because there was such big difference in the Ns between the two groups. Smaller Ns usually lead to larger variances. We conducted a survey to receive narrative feedback however chose not to report the data due to low response. Conclusion Live trainees performed significantly better than online trainees in writing SMART learning objectives. For the whole class, the section on the SMART objectives with the lowest average score was “measurable” followed by “specific”. In spite of the limitations of the study, it is clear that students are capable of incorporating principles of CPD prior to and during their introductory professional practice experiences after focused training (live or online). Future plans include sharing the results with the students and faculty and revising the process of educating the students on how to write SMART learning objectives, continuing the CPD process during didactic months and providing a plan prior to students’ Community Introductory Professional Practice Experience (C-IPPE), assessing improvement in class ability to write learning objectives, and re-deployment of the survey to the same cohort of students as they progress through the pharmacy curriculum. Overall, the faculty was pleased with the implementation of the program and feedback received from the few students who responded to the survey was positive. Our vision of how the CPD process will beimplemented throughout the pharmacy curriculum isexpressed in Figure 2. References1. Rouse MJ. Continuing professional development inpharmacy. Am J Health Syst Pharm. 2004;61(19):2069-2076.2. Accreditation Council for Pharmacy Education.Continuing professional development (CPD). Accessed at http://www.acpe-accredit.org/ceproviders/CPD.asp accessed August16, 2010.3. Tofade TS, Foushee LL, Chou SY, et.al. Evaluation of a Condensed Training Program to Introduce theProcess of Continuing ProfessionalDevelopment. [published online ahead of print Sept13, 2010]. J Pharm Pract. Dec 2010;23(6):560-5694. Austin Z, Marini A, and Desroches B. Use of alearning portfolio for continuous professionaldevelopment: A study of pharmacists in Ontario(Canada). Pharm Educ. 2005; 5; (3/4):175-181.5. Austin Z, Croteau D, Marini A, and Violato C.Continuous professional development: The Ontarioexperience in professional self-regulation throughquality assurance and peer review. Am J PharmEduc. 2003; 67(2): Article 56.6. Gifford, A., Murphy, R. and Anderson, C. (2008) Professional learning and development in pharmacy,Paper presented at the 3rdProfessional LifelongLearning Conference, University of Leeds, June 24,2008.http://www.leeds.ac.uk/medicine/meu/lifelong08/papers/Alison%20Gifford.pdf accessed July 20087. Power A, Johnson BJ, Diack HL, et al. Scottishpharmacists’ views and attitudes towards continuingprofessional development. Pharm World Sci. 2008;30: 136-1438. Pharmaceutical Society of New Zealand Inc. What is ENHANCE. Accessed athttp://www.psnz.org.nz/public/enhance/what_is_enhance/Enhance.aspx August 25, 2010. 9. Pharmaceutical Society of Australia. 2010 Guide toPSA Programs. Accessed athttp://www.psa.org.au/site.php?id=5407 accessedAugust 24, 2010.10. Dopp AL, Moulton JR, Rouse MJ, and Trewet CB. Afive-state continuing professional development pilot program for practicing pharmacists. Am J PharmEduc. . 2010; 74(2) Article 28. Original ResearchEDUCATION http://z.umn.edu/INNOVATIONS2011, Vol. 2, No. 2, Article 40INNOVATIONS in pharmacy711. Bellanger RA and Shank TC. Continuing professional development in Texas: Survey of pharmacists’knowledge and attitudes: 2008. J Am Pharm Assoc.2010; 50(3): 368-374.12. Tofade T, Chou S, Foushee L, et.al. Continuingprofessional development training program amongpharmacist preceptors and non preceptors. J AmPharm Assoc. 2010;50(6):730–735. .13. Accreditation Council for Pharmacy Education.Accreditation standards and guidelines for theprofessional program in pharmacy leading to the doctor of pharmacy degree: Effective July 1, 2007.Accessed at http://www.acpe-accredit.org/pdf/ACPE_Revised_PharmD_Standards_Adopted_Jan152006.pdf accessed August 24, 2010. 14. Fjortoft N. Self assessment in pharmacy education.Am J Pharm Educ. 2006; 70(3) Article 64.15. Rees JA, Cantrill JA, Morris M, et al. Attitudes toreflective practice and continuing professionaldevelopment. Int J Pharm Pract. 2003; 11(S1): R34.16. Wallman A, Lindblad AK, Hall S, et al. Acategorization scheme for assessing pharmacystudents’ levels of reflection during internships. AmJ Pharm Educ. 2008; 72(1): 1-10.17. Janke KK. Continuing professional development:Don’t miss the obvious. Am J Pharm Educ. 2010;74(2) Article 31.18. Fordis M, King JE, Ballantyne CM, et al. Comparisonof the Instructional Efficacy of Internet-Based CME With Live Interactive CME Workshops: ARandomized Controlled Trial. JAMA.2005;294(9):1043-51.19. Sly JL, Lombardi E, Kusel M, Sly PD. Piloting a web-based continuing professional development programfor asthma education. Int J Med Inform 2006; 75(10-11): 708-713.20. Peroz I, Beuche A, Peroz N. Randomized controlledtrail comparing lecture versus self studying by anonline tool. Med Teach 2009; 31(6): 508-512.21. Edginton A, Holbrook J. A Blended Learning Approach to Teaching Basic Pharmacokinetics andthe Significance of Face-to-Face Interaction. Am JPharm Educ 2010; 75(5) Article 88. 22. Landau V. Rubric: Guidelines for evaluatingbehavioral objectives. Accessed athttp://www.roundworldmedia.com/cvc/module4/4nrubric.htm accessedAugust 17, 2010. Original ResearchEDUCATION http://z.umn.edu/INNOVATIONS2011, Vol. 2, No. 2, Article 40INNOVATIONS in pharmacy8Figure 1. Distribution of grades for the online trainees, live trainees and the whole class Original ResearchEDUCATION http://z.umn.edu/INNOVATIONS2011, Vol. 2, No. 2, Article 40INNOVATIONS in pharmacy9Figure 2. Proposed process for incorporating CPD into the pharmacy curriculum Proposed CPD Process Introduce CPD Falland Spring didacticsessions PY 1 Reflect andconduct aneducation plan Act on plan duringsummer IPPEexperienceEvaluate andreflect on summerexperienceConduct plan forPY 2 Didactic yearAct on plan duringPY 2 didactic yearEvaluate andReflect on learningend of PY2 Fall andspring sessions Constructeducation plan for2nd summerrotation Act on plan during2nd summer IPPEexperience Repeat above process for subsequent years Original ResearchEDUCATION http://z.umn.edu/INNOVATIONS2011, Vol. 2, No. 2, Article 40INNOVATIONS in pharmacy10Table 1. Rubric for evaluating learning SMARTaobjectives Objectives are... Satisfactory(5 points)Work in Progress(3 points)Unacceptable(1 point) SpecificIncludes specific informationabout what the learner willbe able to do, e.g. how well,how many, to what degreeObjectives are too generaland don't include specificinformation on what thelearner will be able to do,e.g. how well, how many, to what degreeObjectives don'tdescribe what thelearner will be able to doincludes lists of tasksonly MeasurableObjectives are measurableand use Bloom’s taxonomyof verbs throughoutObjectives use non-measurable verbs, e.g.,understand, learnObjectives use no verbsand include only lists oftasks AchievableThe objectives can beachieved during the timeframe statedSome but not all objectivescan be achieved during thetime frame statedObjectives cannot beachieved during the timeframe stated RelevantObjectives are Relevant toboth the area and level ofpractice of the individualObjectives are Relevant toeither the area or level ofpractice of the individualObjectives are neitherapplicable to the area orlevel of practice of theindividual TimedThe learning objectives haveboth a start and stop date includedLearning objective haseither a start or stop dateLearning objective hasneither a start or stop